Evaluation Methods for Automatic Speech Summarization

We have proposed an automatic speech summarization approach that extracts words from transcription results obtained by automatic speech recognition (ASR) systems. To numerically evaluate this approach, the automatic summarization results are compared with manual summarization generated by human subjects through word extraction. We have proposed three metrics, weighted word precision, word strings precision and summarization accuracy (SumACCY) based on a word network created by merging manual summarization results. In this paper, we propose a new metric for automatic summarization results, weighted summarization accuracy (WSumACCY). This accuracy is weighted by the posterior probability of the manual summaries in the network to give the reliability of each answer extracted from the network. We clarify the goal of each metric and use these metrics to provide automatic evaluation results of the summarized speech. To compare the performance of each evaluation metric, correlations between the evaluation results using these metrics and human judgment are measured. It is confirmed that WSumACCY is an effective and robust measure for automatic summarization.